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ABSTRACT.  Trident  Warrior  2003  (TWOS)  was  a  Navy  initiative  to  demonstrate  an  initial 
baseline  of  potential  FORCEnet  eapabilities.  FORCEnet  is  the  operational  eonstruct  and 
arehitectural  framework  for  Naval  Network  Centric  Warfare  in  the  information  age  that 
integrates  warriors,  sensors,  networks,  command  and  control,  platforms,  and  weapons  into  a 
distributed  combat  force.  TWOS  provided  an  integrated  prototype  capability  for  fleet  evaluation 
and  refinement  of  a  supportable  incremental  delivery  of  FORCEnet  capability.  The  Chat  system 
was  limited  by  the  synchronous  nature  of  the  system  that  required  constant  attention  to  monitor 
communications,  by  the  number  of  participants  that  could  be  accommodated  and  recognized,  and 
by  the  time  required  for  users  to  authorize,  compose,  and  type  messages.  The  connections 
between  the  fire  control  systems  allowed  users  to  share  common  situation  awareness  on  tracks, 
targets,  and  fire  schedules  but  were  mediated  by  the  GCCS-M  position  information,  which  could 
lag  up  to  15  minutes  behind  real-time.  The  utility  of  the  links  between  the  fire  support  systems 
were  limited  by  the  inability  of  one  system  to  accommodate  the  same  target  designations  from 
another  system  and  by  the  lack  of  connection  between  systems.  Situation  awareness  is  a 
continuing  process  and  the  limitation  of  reliance  on  chat  as  a  status  indicator  was  highlighted 
when  one  shooter  was  not  aware  that  he  was  supposed  to  be  in  position  to  provide  fire  support  to 
shore.  Chat  technology  was  used  extensively  to  transfer  information  among  distributed  teams. 
Confusions  and  missed  messages  were  noted  occasionally  and  were  typically  due  to  user 
interface  design  problems,  ambiguous  operating  procedures,  or  technical  incompatibilities 
between  chat  systems.  Display  configurations  and  workspace  layouts  were  problematic  and  led 
to  inefficiencies  in  the  way  that  information  was  transferred  within  and  between  command 
centers.  Consideration  of  the  proper  location  of  operator  workstations,  legibility  of  shared 
displays,  and  easy  access  to  task-relevant  information  would  improve  operations.  While  training 
was  available  on  how  to  operate  individual  FORCEnet  systems,  little  instruction  was  provided 
concerning  how  to  employ  the  systems  for  maximal  operational  effectiveness,  eated  insufficient 
manpower  for  the  new  systems,  limited  their  usefulness  and  adaptability.  HSI  is  an  important 
consideration  in  FORCEnet  systems  analysis  and  assessment.  FORCEnet  systems  rely  upon  the 
performance  of  human  operators  and/or  maintainers,  despite  their  level  of  automation. 

Therefore,  HSI  issues  need  to  be  examined  along  with  the  technical  aspects  of  the  systems 
themselves  as  part  of  the  total  systems  engineering  approach. 
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INTRODUCTION.  Trident  Warrior  2003  (TWOS)  was  a  Navy  initiative  to  demonstrate  an 
initial  baseline  of  potential  FORCEnet  capabilities.  FORCEnet  is  the  operational  construct  and 
architectural  framework  for  Naval  Network  Centric  Warfare  in  the  information  age  that 
integrates  warriors,  sensors,  networks,  command  and  control,  platforms,  and  weapons  into  a 
distributed  combat  force.  TWOS  provided  an  integrated  prototype  capability  for  fleet  evaluation 
and  refinement  of  a  supportable  incremental  delivery  of  FORCEnet  capability. 

The  TWOS  FORCEnet  capabilities  supported  operational  objectives  in  the  following  areas: 

(a)  Expeditionary,  multi-tiered  weapon  and  sensor  information  (Call  For  Fires), 

(b)  Distributed,  collaborative  command  and  control  (C2  /  Collaboration),  and 

(c)  Dynamic,  multi -path  and  survivable  networks  (Network  Operations). 

This  paper  presents  major  HSI  issues  that  were  identified  through  the  analysis  of  field 
performance  data. 

FORCENET  PROCESSES  AND  TECHNOLOGIES 

Call  For  Fires.  Call  For  Fires  furnished  a  context  for  assessing  the  new  form  of  C2  advocated 
by  FORCEnet.  Call  For  Fires  entails  a  coordinated  effort  to  identify,  validate,  and  verify  a 
potential  target  for  fire.  A  major  concern  during  Call  For  Fires  activities  is  to  confirm  that  the 
target  is  hostile,  so  as  to  prevent  friendly  fire  casualties.  To  achieve  its  objectives.  Call  For  Fires 
requires  the  coordinated  use  of  three  technologies:  Automated  Deep  Operations  Coordination 
System  (ADOCS),  Advanced  Field  Artillery  Tactical  Data  System  (AFATDS),  and  Naval  Fire 
Control  System  (NFCS). 

Command  and  Control/Collaboration.  Command  and  control  was  integral  to  all  TWOS 
activities.  It  consisted  of  capabilities  that  facilitated  decision  making,  such  as  acquiring  and 
maintaining  situation  awareness  and  a  common  operational  picture.  Collaboration  enables  the 
sharing  of  information  among  persons  working  on  a  common  task  or  project.  The  collaborative 
technologies  implemented  for  TWOS  were  intended  to  supply  real-time  communication  among 
geographically  separated  individuals  and  groups.  Several  technologies  were  used  that  enabled 
participants  to  acquire  and  maintain  a  shared  awareness  of  the  battlespace:  FORCEview,  Global 
Command  and  Control  System-Maritime  (GCCS-M),  Task  Force  Web  (TFW),  Web  Common 
Operational  Picture  (WebCOP),  and  Collaboration  Tools  (Chat  and  MS  NetMeeting) 

Network  Operations.  Network  Operations  refer  to  establishing,  operating,  and  maintaining 
information  technology  (IT)  networks  that  support  the  activities  of  TWOS.  Hardware  is  the 
primary  focus  of  Network  Operations;  the  four  technologies  that  were  used  during  TWOS  - 
Automated  Digital  Network  Switch  (ADNS),  HF  ALE,  Intra-Battle  Group  Wireless  Network 
(IBGWN),  and  Super  High  Frequency  /  Commercial  Wideband  Satellite  Program  (SHF/CWSP) 

-  did  not  require  ongoing  user  input.  Information  management  personnel  monitored  the 
networks  and  serviced  them  as  needed  throughout  TWOS.  The  Network  Operations  data 
collected  during  TWOS  centered  on  two  activities:  bringing  the  technology  online  and 
maintaining  its  function. 

HUMAN  SYSTEMS  INTEGRATION  (HSI)  IN  NAVAL  OPERATIONS.  HSI  is  a 

comprehensive  management  and  technical  strategy  to  integrate  human  considerations  early  in  the 
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system  design,  development,  and  demonstration  process.  HSI  plays  an  important  role  in  efforts  to 
create  systems  that  accommodate  human  sensory,  perceptual,  cognitive,  and  physical 
performance  characteristics.  Its  major  goals  are  to  improve  total  system  performance  and  reduce 
costs  of  ownership.*  Failure  to  take  HSI  into  account  during  system  design  and  implementation 
typically  results  in  systems  that  are  difficult  to  learn  and  operate  reliably  and  efficiently.  HSI 
accomplishes  its  goals  by  considering  seven  elements  associated  with  system  design, 
development,  and  implementation:  manpower,  personnel,  training,  human  factors  engineering, 
safety,  health  hazards,  and  survivability.  Together,  these  elements  define  how  human  users  affect 
a  system  (in  terms  of  effectiveness,  operation,  and  support  and  their  associated  costs)  and  how  a 
system  affects  the  humans  who  interact  with  it  (e.g.,  operators,  maintainers,  supporters,  trainers). 
Although  each  element  is  important.  Human  Factors  Engineering  has  primary  responsibility  for 
establishing  HSI  human  performance  objectives,  thresholds,  and  characteristics.  This,  plus 
manpower,  personnel,  and  training,  are  the  primary  areas  of  HSI  analysis  for  IT  systems,  such  as 
FORCEnet. 

FOCUS  AREAS  FOR  HSI  ANALYSIS.  In  order  to  assess  the  three  FORCEnet  processes 
(Call  For  Fires;  C2  /  Collaboration;  Network  Operations)  and  their  component  technologies 
during  TWOS,  it  is  necessary  to  define  HSI  analysis  goals. 

An  overarching  goal  of  the  HSI  assessment  in  TWOS  was  to  improve  the  kill  chain.  Technology 
and  the  warfighter  are  the  two  major  elements  in  the  kill  chain  that  provide  the  greatest 
opportunity  to  accelerate  the  decision  and  fire  control  process  from  the  initial  stages  of  target 
detection  to  follow-up  damage  assessments.  Improving  the  kill  chain  depends  on  effective  HSI 
among  FORCEnet  capabilities  and  improved  organizational  processes  and  procedures.  Four 
areas  have  been  identified  as  primary  attributes  of  warfighting  effectiveness  and  served  as  focus 
areas  for  HSI  analysis  in  TWOS: 

•  Shared  Awareness.  A  common  perception  and  understanding  of  the  tactical 
battlespace  and  of  the  roles,  responsibilities,  and  actions  of  other  warfighters. 

•  Efficiency  of  Asset  Utilization.  Length  of  time  needed  to  assign  an  asset,  time 
needed  to  complete  a  mission,  number  of  tasks  accomplished. 

•  Speed  of  Command.  Time  from  when  an  event  occurred  until  the  ordered  action  was 
completed. 

•  Adaptability.  The  extent  and  speed  of  an  organization’s  change  in  response  to 
changing  tactical  situations. 


To  define  these  warfighting  attributes  in  terms  of  human  performance  variables  that  could  be 
measured  during  TW03,  five  HSI  analytic  elements  were  used: 

•  Performance.  The  performance  variable  assessed  shared  awareness  and  speed  of 
command.  Performance  evaluations  were  based  on  three  types  of  performance  data 


*  Department  of  Defense  (January  2001).  Glossary:  Defense  Acquisition  Acronyms  and  Terms  (10*  ed.).  Department 
of  Defense,  Defense  Systems  Management  College.  Fort  Belvoir,  VA. 
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collected  from  TWOS  participants  during  and  after  task  performance:  situation  awareness, 
accuracy,  and  latency  (time  to  perform). 

•  User  interface.  Experienced  human  factors  usability  analysts  conducted  user  interface 
evaluations  based  on  human  factors  usability  heuristics.  These  evaluations  supplemented 
the  performance-based  data  with  more  detailed  analytic  information.  The  results  of  these 
evaluations  are  relevant  to  efficiency  of  asset  utilization,  shared  awareness,  and  speed  of 
command. 

•  Information  transfer.  The  extent  to  which  information  needed  to  operate  each 
technology  in  its  intended  manner  as  well  as  to  achieve  mission  goals  was  assessed.  This 
variable  is  related  to  shared  awareness  and  speed  of  command. 

•  Training.  Training  assessments  were  performed  to  determine  the  type  and  extent  of 
instruction  needed  to  operate  each  technology  effectively,  efficiently,  and  safely;  training 
is  primarily  related  to  adaptability  to  changing  conditions. 

•  Manpower  and  personnel.  Manpower  and  personnel  requirements  for  the  proper 
operation  and  maintenance  of  each  technology  were  determined  to  the  extent  possible 
during  TWOS.  This  variable  is  primarily  related  to  efficiency  of  asset  utilization. 

Together,  these  five  HSI  elements  furnished  the  foundation  needed  to  formulate  and  implement 
an  analytic  plan  that  enabled  meaningful  HSI  assessments  of  the  technological  systems  used 
during  TWOS  in  support  of  FORCEnet  objectives. 

ASSESSMENT  OF  HSI  PROCESSES.  HSI  observers  were  embarked  onboard  two  ships  as 
part  of  an  Expeditionary  Strike  Group  (ESG)  prior  to  and  during  TWOS.  They  collected  data 
during  key  events  related  to  (a)  expeditionary,  multi-tiered  weapon  and  sensor  information  in  the 
‘call-for-fire’  process,  (b)  distributed,  collaborative  command  and  control,  and  (c)  dynamic, 
multi-path  and  survivable  networks. 

HSI  data  were  collected  using  a  range  of  methods,  including  questionnaire  ratings  by  FORCEnet 
users,  interviews  with  operators,  observations  by  HSI  experts  of  operational  performance  and 
system  administration,  training  effectiveness  analysis,  and  usability  analyses  of  FORCEnet 
software  and  workspace  ergonomics. 
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HSI  FINDINGS.  The  following  table  summarizes  the  FORCEnet  operational  processes 
analyzed  during  Trident  Warrior  2003  in  terms  of  major  elements  in  Human  Systems  Integration 
(HSI)^: 


HSI  Process 

Performance 

User 

Interface 

HSI  Element 

Information 

Transfer 

Training 

Manpower  & 
Personnel 

Call  For  Fires 

(D 

A 

A 

A 

C2  /  Collaboration 

A 

A 

A 

A 

Network 

Operations 

(g) 

A 

DISCUSSION.  Individuals  in  the  battle  group  are  often  isolated  and  must  actively  collect  and 
share  information  to  coordinate  their  actions.  Coordination  information  was  passed  between 
distributed  individuals  via  direct  linkages  between  the  fire  support  systems  (ADOCS,  AFATDS, 
NFCS,  GCCS)  and  by  way  of  collaboration  tools  (Chat,  NetMeeting).  The  Chat  system  was 
limited  by  the  synchronous  nature  of  the  system  that  required  constant  attention  to  monitor 
communications,  by  the  number  of  participants  that  could  be  accommodated  and  recognized,  and 
by  the  time  required  for  users  to  authorize,  compose,  and  type  messages.  The  connections 
between  the  fire  control  systems  allowed  users  to  share  common  situation  awareness  on  tracks, 
targets,  and  fire  schedules  but  were  mediated  by  the  GCCS-M  position  information,  which  could 
lag  up  to  15  minutes  behind  real-time. 

The  utility  of  the  links  between  the  fire  support  systems  were  limited  by  the  inability  of 
AFATDS  to  accommodate  the  same  target  designations  as  ADOCS  and  by  the  lack  of 
connection  between  the  NFCS  and  the  shooters  weapon  systems.  These  problems  were 
circumvented  by  the  operators  who  entered  incorrect  data  into  the  systems  that  allowed  the 
support  systems  to  be  used  locally,  but  had  the  unfortunate  result  of  sharing  the  incorrect  data 
with  other  members  of  the  CFF  chain  who  then  assumed  that  specified  targets  had  not  been 
engaged  and  issued  redundant  engagement  orders. 

Situation  awareness  is  a  continuing  process  and  the  limitation  of  reliance  on  chat  as  a  status 
indicator  was  highlighted  when  one  shooter  was  not  aware  that  he  was  supposed  to  be  in  position 
to  provide  fire  support  to  shore.  Design  of  a  tool  that  provides  support  to  users  in  the  form  of 
process  status  indicators  that  augment  the  collaborative  message  content  would  help  avoid  some 
of  these  situation  awareness  problems. 


2(^ 

Fully  functional.  Meets  requirements  but  can  be  improved  with  minor  modifications. 
^  Functional  but  requires  substantial  modifications. 

®  Inadequate  data  were  available  for  valid  assessment. 
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Chat  technology  was  used  extensively  to  transfer  information  among  distributed  teams. 
Confusions  and  missed  messages  were  noted  occasionally  and  were  typically  due  to  user 
interface  design  problems,  ambiguous  operating  procedures,  or  technical  incompatibilities 
between  chat  systems. 

Display  configurations  and  workspace  layouts  were  problematic  and  led  to  inefficiencies  in  the 
way  that  information  was  transferred  within  and  between  command  centers.  Consideration  of 
the  proper  location  of  operator  workstations,  legibility  of  shared  displays,  and  easy  access  to 
task-relevant  information  would  improve  operations. 

Engagement  timelines  for  four  CFF  events  have  been  reconstructed  from  ADOCS  electronic 
logs,  IRC  chat  logs  and  observer  notes.  The  reconstructed  timelines  for  these  4  CFF  events  show 
execution  times  from  31  to  230  minutes.  The  230-minute  response  was  a  difficult  target  with 
high  potential  for  collateral  damage.  But  the  timing  suggests  that  the  distributed  CFF  task 
performers  did  not  realize  that  the  window  for  engagement  had  passed  or  that  a  potential  blue- 
on-blue  situation  might  have  occurred  between  the  NET  and  actual  fires. 

Another  speed  of  command  issue  related  to  the  information  passed  between  members  of  the 
distributed  fire  support  team  was  noted  when  the  fire  planning  was  completed  and  de-conflicted, 
but  the  TAO’s  involvement  in  adapting  to  a  lost  chat  link  prevented  him  from  authorizing  a 
pending  CFF.  The  interconnections  between  the  fire  support  systems  (AFATDS-NFCS-ADOCS- 
GCCS)  provided  the  potential  to  execute  the  fires  scenarios  much  more  safely  and  rapidly,  but 
were  not  complete  enough  to  pass  all  the  information  required  to  automate  the  CFF  process 
(AFATDS  did  not  recognize  TGTD  target  nominations,  and  NFCS  was  not  connected  to  the 
shooter’s  weapons). 

While  training  was  available  on  how  to  operate  individual  FORCEnet  systems,  little  instruction 
was  provided  concerning  how  to  employ  the  systems  for  maximal  operational  effectiveness. 
Because  of  the  concurrent  events  during  TW03,  trained  operators  and  maintainers  were  quite 
limited  for  most  FORCEnet  systems.  This  created  insufficient  manpower  for  the  new  systems, 
limited  their  usefulness  and  adaptability. 

CONCLUSIONS.  HSI  is  an  important  consideration  in  FORCEnet  systems  analysis  and 
assessment.  FORCEnet  systems  rely  upon  the  performance  of  human  operators  and/or 
maintainers,  despite  their  level  of  automation.  Therefore,  HSI  issues  need  to  be  examined  along 
with  the  technical  aspects  of  the  systems  themselves  as  part  of  the  total  systems  engineering 
approach.  Specific  HSI  issues  were  identified  for  FORCEnet  technologies  and  for  their 
integration  with  legacy  systems  and  procedures.  Many  of  the  FORCEnet  technologies 
functioned  well  in  TW03,  but  several  areas  for  potential  HSI  enhancements  were  uncovered 
during  the  exercise. 

Structured  exercise  test  procedures  are  needed  to  permit  collecting  quantitative  human 
performance  data,  including  situation  awareness,  decision  latency,  and  workload.  HSI  data 
collection  and  analysis  was  constrained  by  the  diverse  and  often  competing  objectives  of  TW03, 
by  the  unstable  introduction  of  scenario  events  /  MSELs,  and  by  the  uncertainty  about  how  the 
technologies  would  be  used.  Efforts  to  address  these  methodological  issues  earlier  during 
exercise  planning  would  permit  much  greater  precision  in  the  HSI  data  collected. 
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Human  Systems  Integration 

(HSI) 


Integrates  human  capabilities  and 
limitations  into  system  definition, 
design,  development,  and 
evaluation  to  optimize  human- 
system  performance  under 
operational  conditions 


Operational  Environment 


Warfighter 


Mission 

Tasks 


Dispiays/ 

Interfaces 


Part  of  total  systems 
engineering  approach  to 
anaiysis,  design, 
deveiopment,  and  test  of  a 
product  or  system 


US  Navy  Network  Centric  Warfare  (NCW): 
^  ^  JFORCEnet 


“FORCEnet  is  the  operational 
construct  and  architectural 
framework  for  Naval  Warfare  in  the 
Information  Age  which  integrates 
WARRIORS,  sensors,  networks, 
command  and  control,  platforms 
and  weapons  into  a  networked, 
distributed  combat  force,  scalable 
across  the  spectrum  of  conflict 
from  seabed  to  space  and  sea  to 
land.”  * 

*  CNO’s  Strategic  Study  Group  -  XX  definition 

from  22  July  02  CNO  Briefing 


“Currently,  FORCEnet  is  not  on  track.  It  needs  to  become  more  warrior  centered, 
assume  an  enterprise-wide  culture,  and  develop/apply  metrics.”  ssg-xxii,  juiy  2004 


“The  real  need  (in  FORCEnet)  is  to  examine  Decision  Making,  Cognition,  and  C2  -  the  human 
element.  The  current  FORCEnet  capabilities  packages  are  too  narrow  ...  The  focus  should 

be  on  increased  speed  of  action,  distributed  forces,  and  goals . need  to  examine  the 

alignment  of  requirements,  resources,  and  providers  to  create  mutually  reinforcing 
technological  requirements 

>  Evaluate  system  of  systems  performance  (an  integration  of  factors  and  dimensions) 

>  Assess  effective  engagement  and  operations  (human  understanding,  decision  making,  and  C2) ...” 


^  FORCEnet  Assessment  Continuum 
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Trident  Warrior  03: 

A  initiai  Demonstration  of  NCW 


GCCS-M 

EMAIL 
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GCCS-M 
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FORCEview 
EMAIL  ^ 
CHAT 
VTC 


.DSCS 
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HSI  Objectives 

•  Establish  HSI  requirements  and  identify  HSi  deficiencies  for 
FORCEnet  technoiogies 

-  Shared  Situation  Awareness.  A  common  perception  and 
understanding  of  the  tactical  battlespace  and  of  the  roles, 
responsibilities,  and  actions  of  other  Warfighters 

-  Efficiency  of  Asset  Utilization.  Length  of  time  needed  to  assign 
an  asset,  time  needed  to  compiete  a  mission,  number  of  tasks 
accomplished 

-  Speed  of  Command.  Time  from  when  an  event  occurred  until  the 
ordered  action  was  completed 

-  Adaptability.  The  extent  and  speed  of  an  organization’s  change 
in  response  to  changing  tactical  situations 
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HSI  Measures 


Performance 

Task  Performance 

Situation  Awareness  (SA) 

Decision-Making  (Speed  of  Command) 

Usability 

Ease  of  Use 

Frequency  and  Context  of  Use 

Scaiability/Taiiorability  of  Displays 

Efficiency  of  Use 

Effectiveness  of  Use 

Error  prevention  and  Handling 

Visibility 

Consistency/Familiarity  of  HCI 

Satisfaction/Preference 

Information 
Transfer  and 
Knowledge 
Management 

Ability  to  Share  Information/Access  Information 

Quality  of  Information  Exchanged  during  Collaboration  Process 

Shared  Understanding  of  Missions,  Roles,  and  Tasks 

Efficiency  of  Information  Exchange 

Effectiveness  of  Information  Exchange 

Reliability/Credibility  of  Information 

Training 

Amount  of  Training  Provided  and  Required 

Value  of  Training 

Proficiency 

Documentation  and  Online  Help 

Manpower 

and 

Personnel 

Manpower  Required  to  Operate/Administer  Systems 

Availability 

Match  between  Personnel  and  System  Requirements 

W  Major  HSI  Findings  in  TW  03 
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Lack  of  a  concept  of  operations  for  FORCEnet  technologies; 
reduced  shared  situation  awareness  and  ability  to  adapt  to 
changing  demands 

User  interfaces  need  improvement:  display  configurations, 
workspace  layouts,  inefficiencies  in  how  information  was 
transferred  within  and  between  command  centers;  legibility  of 
shared  displays,  and  access  to  task-relevant  information 

While  training  was  provided  on  individual  FORCEnet  systems, 
no  instruction  was  available  on  how  to  employ  systems  for 
maximal  operational  effectiveness  resulting  in  operational 
inefficiencies 


Insufficient  manpower  available  for  new  FORCEnet  capabilities 
without  removal  of  legacy  systems 


Trident  Warrior:  Looking  Forward 
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Expeditionary,  multi-tiered  weapon  and  sensor 
information  (Call  For  Fires) 

Distributed,  collaborative  command  and  control 
(C2  /  Collaboration) 

Dynamic,  multi-path  and  survivable  networks 
(Network  Operations) 

HSI  =  “Come  as  you  are” 


-  Networks 

-  Web-Enabled  Warfighter 

-  Sea  Warrior  (Focus  on  Sailor  career) 

-  ISR/fires 


TW-04 


HSI  =  Specific  Objectives 

Shared  Situation  Awareness,  Speed  to 
Command,  Efficient  Use  of  Warfighter 


HSI  =  Real  Opportunities 

•  Multi-Echelon 

•  Split  Staff 

•  Shared  SA  of  Intel 
Analysis 


-  Carrier  Strike  Group 

-  Coalition  Networks 

-  Multi-Level  Security  Systems 

-  Inter-Force  Wireless  Networks 

-  Joint  Distributed  C2 
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Information  Design 
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HSI  -  Example  K-Web  Overview  Page 
(shows  status  across  mission  areas) 


Overview  -  Microsoft  Internet  Explorer 

Knowledge 

Management 

1  File  Edit  View  Favorites  Tools  Help 

.^^Back  0  @  fl!) 

>P  Search  Favorites  Media  H  "  [~J  -  A 

Address 

^  C:\KWeb  Files\KWebUnclass\Detault.htm 
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Assessment  Approach 


CHALLENGES 

•  Functionality  “between”  stovepipes 

-  Design 

-  CONORS 

•  Skills 

-  Required  ievel 

-  individual  vs.  Team 

-  Training/Experience 

•  Workload 

-  Cognitive  and  physical 

-  Manpower 

•  Acquisition 

-  Formal  acquisition  programs 

-  Non-acquisition  programs 

•  Human  performance 

-  Demonstrated  improvements 

_ -  Quantifiable  measures _ 


New 

Technologies/ 

Capabilities 


Concept 
of  Operations 
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Scenario  /  Op  Context 


Mission 

Mission 

Areas 

-  F 


NMETL 


NMETL 


NMETL 

Functions 


"ask 


"ask 


"ask 


Task 

Steps 


HSI  Components 

-  Human-Computer  Interface 

-  Training  &  Technical  Docs 

-  Manpower  &  Personnel 

-  Organizational  Info  Flow 


Outcome 

Measures 


Process 

Measures 


Thin-Thread 

Measures 


APPROACH 


Integrated,  multi-method  assessment 

Takes  advantage  of  available  assessment 
opportunities 

Linked  with  mission,  tasks,  and  jobs 

-  Extends  ongoing  MPT  efforts  in  system  acquisition 

Transferable  across  technologies 

Easily  expanded  to  incorporate  future  developments 
Addresses  all  facets  of  human  performance  in 
complex  systems 

-  Recognizes  complex  relationship  between 
Technology,  HF,  MPT,  etc. 


MOEs 

MOPS 

HSI  Metrics 

Analysis  &  Heuristic  Review 

Focuses  on  technologies  and  system  components 
Based  on  HSI  standards  and  design  guidelines 
JJsability  Tests  &  Limited  Assessments 

-  Examines  technologies  in  simulated  task  context 

-  Empirical  studies  to  get  feedback  from 
warfighters  on  utility,  usability,  reliability,  etc. 

^Scenario-based  Exercises  (IPD) 

.Verifies  technologies’  contribution  to  ME  in 
|dynamic,  operational  context 

-  Observation  of  global  performance  ->2 
indicators 


Future  Directions:  HSI  Enterprise  Architecture 
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Warfighter  Optimization  ■ 

Career  Mgmt  /  Planning 
Growth  &  Development 
Distribution  /  Position  Mgmt 
^  Performance 


Warfighting  System 


Optimal  Manning 

^  Reduced  Total  Ownership  Cost 
^  Performance 
Mission  Readiness 


Design  Optimization 


Capability  Needs 

Concept  Development  1 

Technology  Selection 

System  Development, 

Deployment,  Operations 

&  Solution  Definition  ; 

&  Refinement  ; 

&  Development 

Demonstration  and  Production 

&  Support 

Executable  Architecture  Framework 

Ontology  Tool 

- ► 

Application  Framework 

— ► 

Code  Generation 

- ► 

Deployment 

Platform  Independent  ◄- 


Architecture  Metadata 


■>  Platform  Specific 


Services  &  Tools 


HSI  Execution 
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Design  Optimization 


Acquisition  Spiral 


Capability  Needs  1 

I 

Concept  Development  1 

Technology  Selection 

System  Development, 

Deployment,  Operations 

&  Solution  Definition  ; 

&  Refinement  ; 

&  Development 

Demonstration  and  Production 

&  Support 

Conclusions 


*  The  Warfighter  is  a  criticai  component  of  Network 
Centric  Warfare  (NCW) 

*  HSI  Assessment  in  TW04  and  other  NCW  events 

-  Comprehensive  HSI  data  collection  and  analysis 

-  Comparisons  to  TW03  and  other  exercises  to  see  trends 

-  Aggregation  across  FORCEnet  tests  to  determine  HSI  impacts 

*  Greater  focus  on  development  of  TTP  /  ConOps 

*  Standard  set  of  HSI  measures  and  metrics 

*  Predictive  analysis  of  HSI  issues  for  new  systems, 
design  options,  manning,  training,  etc. 


HSI  Enterprise  Architecture  to  structure  HSI 
performance  taxonomy  and  data  repository 
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FORCEnet  Processes  and 
_  Technologies 


■  •  Call  For  Fires 

-  Automated  Deep  Operations  Coordination  System 
(ADOCS) 

-  Advanced  Fieid  Artiiiery  Tacticai  Data  System  (AFATDS) 

-  Navai  Fire  Controi  System  (MFCS) 

•  Command  and  Control  /  Collaboration 

-  FORCEview 

-  Global  Command  and  Control  System-Maritime  (GCCS-M) 
-  Task  Force  Web  (TFW) 

-  Web  Common  Operational  Picture  (WebCOP) 

-  Collaboration  Tools  (Chat  and  MS  NetMeeting) 

•  Network  Operations 

-  Automated  Digital  Network  Switch  (ADNS) 

-  High  Frequency  Automatic  Link  Establishment  (HF  ALE) 

-  Intra-Battle  Group  Wireless  Network  (IBGWN) 

-  Super  High  Frequency  /  Commercial  Wideband  Satellite 
Proaram  fSHF/CWSP^ 
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Call  For  Fires  -  1 


Performance 

•  Very  limited  shared  SA  among  Fiag  Piot,  SACC,  and  LFOC. 

•  SACC 

-  Drawbacks:  did  not  hold  BDA  on  targets  once  the  target  was  issued  to  a  Fires 
resource.  Flag  Plot  did  not  know  about  every  target  being  taken  until  SACC 
asked  for  BDA. 

•  ADOCS:  Supplies  adequate  amount  of  task-reievant  information 

-  Drawbacks:  very  slow  when  running  other  software,  including  C2PC. 

•  AFATDS:  Works  weli  if  operator  is  weil  trained:  maintains  COP, 
deconfiicts  fires,  and  has  good  interoperability.  Simplifies  identification  of 
force  locations  and  status. 

-  Drawbacks:  a  need  to  enter  database  and  IP  addresses  and  slow  processing 
speed. 

•  MFCS:  Presents  information  so  it  can  be  readiiy  understood. 

User  Interface 

•  ADOCS:  No  specific  usability  problems  were  noted.  Displays  are  easy  to 
read. 

•  AFATDS:  The  better  trained,  frequent  users  rated  AFATDS  as  very  usable. 
Displays  were  easy  to  read,  and  information  was  presented  logicaiiy. 

-  Drawbacks:  limited  error  prevention  and  detection,  inconsistent  color-coding, 
use  of  non-standard  commands,  and  incomplete  user  feedback. 


Call  For  Fires  -  2 


Information  Transfer 

•  Technical  integration  across  systems  for  fires  process  worked  very  well. 

-  Drawbacks:  Problems  observed  in  transferring  information  between  SACC 
and  ESG.  Only  way  to  exchange  information  between  SACC,  LFOC,  and  Flag 
Plot  was  via  Chat. 

•  Time  Sensitive  Targeting  messages  were  often  sent  directly  from  JIC  to 
SACC. 

-  Drawbacks:  Flag  Plot  was  not  always  aware  of  TST  in  progress. 

•  ADOCS:  2/3  of  users  felt  that  ADOCS  did  not  enhance  SA  and  did  not 
help  to  identify  force  iocations,  understand  status  of  users  and 
equipment,  and  track  tasking  and  scheduling. 

-  Drawbacks:  Problems  noted  in  understanding  status  of  other  users  and  their 
equipment,  estimating  opposing  forces  capabilities,  anticipating  responses  to 
Blue  actions,  identifying  and  resolving  scheduling  conflicts,  time  lag  between 
SA  and  real  events,  and  shared  SA  among  team  members. 

•  AFATDS:  Supported  SA  and  helped  to  identify  force  locations, 
understand  status  of  users  and  equipment,  and  track  tasking  and 
scheduling. 

-  Drawbacks:  Potential  SA  problems  noted  in  monitoring  critical  events; 
resolving  scheduling  and  resource  conflicts,  and  anticipating  responses  to 
blue  actions. 

•  MFCS:  Presents  information  so  it  can  be  readily  understood. 


19 


Call  For  Fires  -  3 


Training 

•  A  systems  integration  guide  is  needed  to  show  how  individuai  CFF 
technoiogies  reiate  to  each  other  from  an  operationai  (task) 
perspective. 

•  ADOCS:  Training  was  described  as  inadequate.  Procedures  were  not 
weii  understood  and  users  wanted  more  training. 

•  AFATDS:  More  extensive  formai  training  is  needed.  Users  often  did  not 
understand  AFATDS  procedures  or  find  job  aids  usefui.  The  number  of 
requests  for  technicai  assistance  (to  setup  the  system)  was  excessive. 

•  MFCS:  More  training  needed  to  gain  competence  (4  hours  training 
provided). 

Manpower/Personnei 

•  Manpower  was  oniy  one-deep  for  most  CFF  technoiogies. 
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Command  and  Control/Collaboration  - 

1 


Performance 

•  The  Flag  Plot  watch  team  took  14  minutes  to  find  the  right  format  for  an 
NBC-1  Report  and  12  minutes  to  draft  the  message. 

•  The  best  SA  was  gained  through  the  voice  SITREPS. 

•  Chat: 


-  Drawbacks:  Multiple  Chat  tools  caused  confusion.  Several  users  noted  that 
monitoring  several  chat  rooms  increased  their  workload.  Navy  Enterprise 
Portal  helps  to  consolidate  chat  rooms,  but  does  not  work  with  all  chat 
programs. 

•  GCCS-M:  Frequent  users  can  reduce  the  time  needed  to  organize 
information  and  decide  what  actions  to  take. 

•  NetMeeting: 

-  Drawbacks:  Requires  a  lot  of  bandwidth.  Consistent  availability  is  an  issue. 
Whiteboard  slowed  system  down  too  much,  creating  unacceptable  time  lags. 
The  10  user  limit  was  a  serious  shortcoming.  File  transfers  were  rated 
disorganized  and  ineffective. 

•  WebCOP:  Worked  weii;  used  for  briefs  in  Fiag  Plot. 

-  Drawbacks:  Occasionally,  COP  was  lacking:  (a)  COP  in  Flag  Plot  was  did 
not  match  COP  on  the  Chancellorsville;  and  (b)  specific  objects  could  not  be 
located  or  their  map  locations  were  inaccurate. 
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Command  and  Control/Collaboration  - 

2 

User  Interface 

•  Ergonomic  deficiencies  in  the  Flag  Plot  workspace: 

-  High  traffic  and  noise  levels  hindered  task  performance  and  development  of  SA. 

-  Traffic  flow  intermittently  blocked  view  of  large  screen  displays  at  front  of  Flag  Plot. 

-  Legibility  of  text  on  large  screen  displays  at  front  of  Flag  Plot  was  poor  for 
observers  seated  in  the  back  of  Flag  Plot. 

-  Locations  of  some  displays  produced  difficult  viewing  angles. 

•  Technical  incompatibilities  among  the  different  Chat  systems. 

•  Flag  spaces  need  larger  displays  for  watch-standers. 

-  Multiple-screen  console  (e.g.,  Multi-Modal  Watch  Station  or  Knowledge  Desk)  would 
be  helpful. 

•  Chat:  Good  screen  legibility,  information  presentation,  and  navigation. 

-  Drawbacks:  (a)  inadequate  feedback,  (b)  poor  error  prevention  and  recovery,  (c) 
difficulties  in  gaining  SA  without  viewing  entire  chat  sequence  (scrolling  issues),  (d) 
limited  formatting  capabilities,  and  (e)  pressing  the  Enter  key  inadvertently  sends 
Chat  message.  MS  Chat  deficiencies  included  no  authentication,  no  time  stamp, 
and  no  auto  logging. 

•  GCCS-M: 

-  Drawbacks:  non-standard  and/or  inconsistent  icons,  menus,  buttons,  navigation, 
operating  procedures,  and  commands.  Large  amount  of  information  clutters  its 
display.  Inadequate  error  prevention  and  recovery. 

•  NetMeeting: 

-  Drawbacks:  information  presentation,  navigation,  and  error  detection  and  recovery22 


Command  and  Control/Collaboration  - 

3 


Information  Transfer 


•  Much  'sneaker  net'  occurred  between  key  watch  standers  in  the  JiC  and 
LFOC  going  to  and  from  the  Fiag  Piot. 

•  Chat:  Chat  generaiiy  reduced  time  and  effort  needed  to  identify  users, 
exchange  information,  and  support  coordination  among  users. 

-  Drawbacks:  Confusion  occurred  when  orders  sent  via  Chat  were  not 
acknowledged.  Timely  messages  dependent  on  typing  skills  of  sender.  BWC 
had  to  approve  all  chat  dialogue  before  it  was  sent. 

•  GCCS-M:  Generally,  simplified  identifying  force  locations. 

-  Drawbacks:  SA  problems:  (a)  Keeping  track  of  tasking,  scheduling,  and 
critical  events;  (b)  Identifying  scheduling  and  resource  conflicts;  (c)  Tracking 
progress  toward  objectives,  and  (d)  Anticipating  responses  to  blue  actions. 
Some  users  felt  their  SA  lagged  significantly  behind  actual  events. 
(Catastrophic  effect  if  GCCS  is  used  to  provide  track  information  for  fire 
support  systems.) 

•  WebCOP:  Kept  track  of  mission  goals  and  objectives,  critical  events, 
and  goals  and  actions. 


-  Drawbacks:  SA  ratings  were  moderately  negative. 


Command  and  Control/Collaboration  - 

4 


Training 

•  Crew  was  mostly  unaware  of  capabilities  of  Fn  technologies. 

•  A  new  concept  of  operations,  and  staff  familiarity  with  it,  is  needed  within  the 
ESG  to  promote  effective  collaboration  using  the  Fn  technologies. 

•  The  tempo  of  operations  in  Flag  Plot  was  very  high,  due  in  part  to  confusion  over 
the  use  of  the  new  C2  Fn  technologies. 

•  Lack  of  standardized  message  posting  procedures  on  the  ESG  web  site 
complicated  finding  information. 

•  Chat:  Most  users  reported  no  formal  training  on  Chat,  but  only  3  wanted 
additional  training. 

•  GCCS-M: 

-  Drawbacks:  Many  users  did  not  understand  its  procedures,  felt  inadequately 
trained,  and  that  documentation  and  online  help  needed  improvement. 

•  NetMeeting:  Most  felt  they  had  received  adequate  training,  with  half  reporting  that 
documentation  and  online  help  were  all  the  training  needed. 

•  Task  Force  Web: 


-  Drawbacks:  All  users  felt  they  had  not  received  adequate  training  and  that  the 
online  help  was  not  sufficient  training  by  itself. 

Manpower/Personnel 

•  Chat: 


-  Drawbacks:  More  simultaneous  chat  rooms  were  in  use  than  could  be  effectively 
monitored  and  serviced  by  the  assigned  staff. 

•  GCCS-M,  NetMeeting:  The  number  of  calls  for  technical  assistance  was 


Network  Operations 
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Performance 


•  ADNS:  Stable;  no  trouble  calls  reported  during  TW03. 

•  HF  ALE:  The  Marines’  HF  ALE  radios  do  both  voice  and  data. 

-  Drawback:  Shipboard  radios  could  not  handle  data,  voice  only. 

•  SHF/CWSP: 

-  Drawbacks:  SHF  was  difficult  for  the  operators/administrators  to  manage.  If  the 
hardware  is  powered  down,  all  configuration  settings  are  lost,  requiring  increased 
workload  to  reconfigure. 

User  Interface 


•  IBGWN: 

-  Drawbacks:  Very  complex,  non-intuitive  interface.  When  switching  between 
configuration  displays,  different  screens  would  contradict  each  other  for 
established  connections. 

Training 

•  Network  Systems: 

-  Drawbacks:  Training  for  the  new  network  technologies  was  marginal.  Training  for 
most  of  the  new  technologies  was  left  to  the  individual  ships  and  groups.  ITs  were 
not  given  training  on  the  Network  Operations  system  as  a  whole.  Single  technology 
expertise  is  inadequate  since  network  technologies  are  becoming  highly 
interrelated. 

•  IBGWN: 


-  Drawbacks:  No  technical  representatives  available  during  TWOS  for  training  or 
information. 


Manpower/Personnel 
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MOE  Summary 


HSI  Process 

Measure  of  Effectiveness 

Performance 

User 

Interface 

Information 

Transfer 

Training 

Manpower& 

Personnel 

Call  For  Fires 

© 

© 

A 

A 

A 

C2I 

Collaboration 

A 

A 

A 

A 

© 

Network 

Operations 

© 

A 

0  Fully  functional.  Meets  requirements  but  can  be  improved  with  minor  modifications. 
^  Functional  but  requires  substantial  modifications. 

^  Largeiy  non-functionai  and  needs  major  modifications. 

^  Inadequate  data  were  available  for  valid  assessment. 
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MOEs,  MOPs,  and  Metrics  Examples 

KSi _ 

MOE:  Performance 

•  MOPs 

-  Task  Performance:  Extent  of  support  for  required  tasks 

•  Metrics: 

-  Extent  of  usage  of  tool/application  to  support  task 

-  User  rating  of  usefulness  of  tool  to  support  task 

-  User  rating  of  task  efficiency  -  timeliness  to  complete  task 

-  Observer,  SME,  or  superior  rating  of  task  efficiency  -  timeliness  to  complete  task 

-  User  rating  of  task  effectiveness/quality  -  task  accuracy 

-  Observer,  SME,  or  superior  rating  of  effectiveness/quality  -  task  accuracy 

-  User  rating  of  workload  associated  with  performing  task 

-  Situation  Awareness  (SA):  Understanding  of  relevant  aspects  of  operational 
situation,  relationship  between  these  with  each  other  and  evolving  situation, 
and  how  the  situation  and  events  will  unfold  in  the  future 

•  Metrics: 

-  Accuracy  and  timeliness  of  answers  to  questions  embedded  in  scenario  communications 

-  Accuracy  of  answers  to  questions  at  “stopping  points”  of  scenario 

-  Decision-making:  Availability  and  extent  of  use  of  tools  to  support  decision¬ 
making;  timeliness  and  accuracy  of  decision  making  (Speed  of  Command) 

•  Metrics: 

-  Availability  of  tools/application  to  support  decision-making 

-  Use  of  tool/application  to  support  decision-making 

-  Observer,  SME,  or  superior  rating  of  accuracy  and  timeliness  of  decisions 


If  Major  HSI  Findings  in  TW  03 

KSi _ 

•  Loss  of  SA 

-  Displays  did  not  support  Flag  Plot  functions 

-  Poor  workspace  layout  (screen  real  estate) 

-  No  standardization/no  functional  analysis  on  decision  making 
process 

•  Unclear  where  to  find  or  post  info 

•  Many  websites/portals/chatrooms  to  monitor 

•  Impaired  Speed  of  Command 

-  26  minutes  for  NBC  report 

-  8  minutes  to  assess  hostile  intent  of  track 

•  Information  Exchange  Shortcomings 

-  Chat  used  to  pass  orders;  not  always  acknowledged 

•  Little  or  no  system  integration  training  provided  for  new  Fn 
systems 

•  Manpower  analysis  for  new  Fn  systems  not  addressed;  legacy 
systems  remain 

Insufficiently  developed  concept  of  operations  for  integrating  Fn  technologies  with  current 
information  transfer  procedures.  Warfighter  unable  to  fully  understand  how  to  employ  Fn  capabilities,  i 


Situation  Awareness 
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Situation  Awareness 

-  Assembling  information  across  multiple  agents  over  extended  time  ... 
asynchronous,  distributed  environment 

-  Piece  together  scraps  to  form  likely  scenario 
or  course  of  action 

-  Ensure  a  shared  perspective  (and  projection)/ 

Ideal  SA:  The  information  and  knowledge 

requirements  defined  by  experts* 

Achievabie  SA:  the  subset  of  Ideal  SA 
available  to  the  decision  makers* 

Actual  SA:  The  subset  of  Achievable  SA 
inferred  from  measurement  or  observation* 


*Pew,  2000 

Situation  Awareness  (SA)  is 

(1)  the  perception  of  eiements  in  the  environment  within  a  voiume  of  time  and  space, 

(2)  the  comprehension  of  their  meaning,  and 

(3)  their  status  in  the  near  future.  ”  (Endsley,  1988) 


UNDERSTANDING  SA 


Situation  Awareness  (According  to  Pew  (2000)) 


Ideal  SA:  The  information  and 
knowledge  requirements  defined  by 
experts,  often  after  the  fact. 


Achievable  SA:  The  subset  of  ideal 
SA  that  is  available  to  the  decision 
makers. 


Actual  SA:  Inferred  from 
measurement  or  observation  and  is 
a  subset  of  achievable  SA. 
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